Residual modeling in music analysis-synthesis

In analysis-synthesis of musical sounds based on a sinu-soidal model, the diierence between the original signal and the synthesized signal, termed the residual, is typically a broadband noise process. It contains such musical phenomena as ute breath noise or violin bow noise. Synthesis without such \noise" tends to sound artiicial; it is desirable to improve the synthesis realism by modeling the residual in such a way that it can be reinjected in the synthesized signal. This paper deals with a model of noise perception based on the equivalent rectangular bands (ERBs) of the auditory system. Since a broadband noise is perceptually well-represented by the time-varying energy in each of these frequency bands, the residual is parametrized in terms of these energies in the proposed model. An application of the model to music synthesis based on the inverse fast Fourier transform (FFT) is described in detail. 1. SINUSOIDAL MODELING AND ADDITIVE SYNTHESIS In analysis-synthesis, the diierence between the original signal and the synthesized signal is called the residual. The properties of the residual are determined by the underlying signal model and the analysis-synthesis method. A common model for musical signals is a deterministic plus stochastic decomposition where the deterministic component d(t) is a sum of slowly-evolving sinusoids called partials and the stochastic component s(t) is a broadband noise process 1]: d(t) = Q X q=1 Aq(t) cos q(t) (1) Here, Q is the number of partials; Aq(t) describes the time-varying amplitude of the q-th partial and the total phase q(t) describes its frequency evolution and phase ooset. This model is useful for musical applications since the parameters allow for desirable modiications such as pitch-shifting, time-scaling, and a wide variety of spectral transformations such as cross-synthesis. For this signal model, the analysis typically consists of a short-time Fourier transform (STFT) followed by a spectral peak-picking algorithm that tracks the peaks from one analysis frame to the next; such an analysis typically nds the amplitude, frequency, and phase of each partial in each frame 1, 2]. If the partial parameters vary slowly with respect to the analysis frame rate, the frame rate samples can be reliably interpolated in the ensuing synthesis using low-order evolution models such as linear amplitude. Any error in the synthesis interpolation appears in the residual, as do any partials missed by the analysis; such partials can be removed by re-analyzing the residual. More notably, the residual also contains any …